2 Dimensionality Reduction

In many applications today, data is drawn from a high-dimensional feature space, where the dimension d is incredibly high. That is, we view each data point as a vector in Rd. A few examples of high dimensional data: • DNA sequencing: each nucleotide in the sequence is a feature. • Health records: various measurements like weight, blood pressure, diagnosed diseases, medications, nutrition, etc. • Computer vision: every pixel of an image/video is a feature. A problem with high dimensional data is that many algorithms we use to extract higher-order information (clustering, nearest-neighbors, etc.) are severely impacted by high dimension; something like nd in the runtime is not uncommon. There are a few strategies for making high-dimensional data palatable for these algorithms: 1. Restrict to important dimensions (pick some k d dimensions). This is difficult to do in general, since it may not be clear which features are “important” to preserving the core characteristics of the data set.